Papers with sequence processing
Subformer: Exploring Weight Sharing for Parameter Efficiency in Generative Transformers (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Recent improvements in NLP tasks can be attributed to the Transformer model. |
| Approach: | They propose to use parameter-sharing methods to reduce parameter budgets in generative models by using sandwich-style parameter sharing and self-attentive embedding factorization. |
| Outcome: | The proposed model outperforms the current RNN model even with significantly fewer parameters. |
Quantifying Adaptability in Pre-trained Language Models with 500 Tasks (2022.naacl-main)
Copied to clipboard
| Challenge: | a recent study examines the features and limits of LM adaptability to new tasks . many questions about the nature and limits remain unanswered . |
| Approach: | They evaluate adaptability to new tasks using a new benchmark, TaskBench500 . they find adaptation procedures differ dramatically in their ability to memorize small datasets . |
| Outcome: | The proposed benchmark compares 500 procedurally generated sequence modeling tasks to a new benchmark. |